Typically on routers and other networking devices the memory (DRAM) is logically divided into two parts. The first part is the Main Processor Memory, referred to in the following as PMEM, which is the part of the memory that is accessed only by the CPU. The second part is Shared I/O Memory which is accessed by both the CPU and I/O Devices.
The PMEM is used by program TEXT, DATA, BSS, HEAP and STACK. The routing tables, route caches, configurations, etc., will fall into PMEM. Since PMEM is accessed ONLY by the CPU, it is mapped using the CACHE attribute which means that when a memory location in this space is accessed the corresponding line is brought into the cache so that subsequent accesses to that memory location are made from the cache. Cache is a small fast memory located closer to the CPU than the main memory. The accesses to the cache are an order of magnitude faster than the accesses to the main memory. The PMEM is cached with the write-back attribute which means that whenever some modification is made to this memory location by the CPU the change is reflected only in the cache and not in the main memory. The memory is updated only when this cache line in the cache needs to be replaced. The main point to note here is that cacheable memory accesses are much faster than the non-cacheable memory accesses.
I/O Memory, referred to in the following as IOMEM, is the part of the memory that is shared by the CPU and other I/O devices, such as the Network Controllers, and is used by I/O Devices for storage of incoming packets and by the CPU for placing outgoing packets, packet data buffers, particle data buffers, and device control structures (descriptors, address tables, etc.) reside in IOMEM. Packets received on the network interfaces are placed in IOMEM by the I/O devices to be picked up and processed by the CPU. After the processing is done, CPU will place the packets in the IOMEM to be picked up and transmitted by the I/O devices.
Since IOMEM is shared by multiple memory masters, it is mapped with the NON-CACHABLE attribute. That means that any access to this part of the memory goes all the way to the memory and nothing is brought into cache. This is done to preserve coherency of the memory. If IOMEM is made cachable and brought into cache by the CPU, updates made by the CPU will not be made to the memory and thus will not be seen by the I/O devices.
As part of packet processing the CPU will perform multiple accesses to the packet data which lies in IOMEM. These accesses are to: 1) decode the encapsulation; 2) read the Layer 3 addresses; and 3) do other things needed to route the packet.
Since these CPU accesses to IOMEM are all Non-Cached accesses, they are a major performance bottleneck because, as described above, Non-Cached accesses are an order of magnitude slower than Cached accesses.